Metadata in multi image scenes

ABSTRACT

A spatial relationship is determined between a first image of a first region of a physical space, and a second image of a second region of that space; and the first and second images are thereby stitched together into a composite image comprising first and second areas derived from the first and second images respectively. Further, there is detected an embedded signal having been embedded in light illuminating at least part of the first region of the physical space upon capture of the first image, the embedded signal conveying metadata relating to at least part of the physical space. An effect applied to at least part of the first area of the composite image based on the detected metadata; and also, based on the detected metadata and on the determined spatial relationship between the first and second images, the effect is applied to at least part of the second area of the composite image that extends beyond the first area.

TECHNICAL FIELD

The present disclosure relates to the inclusion of metadata in an imageof a scene, and particularly in a composite image created from multipleindividually-captured images stitched together using image processingtechniques.

BACKGROUND

A street viewing service is a service that captures images of thescenery surrounding public roads and the like, in order to make theseimages available online. These services offer a user the ability totraverse through a virtualized view of the real world. Typically imagecapturing devices mounted on cars and potentially other modes of travelsuch as bicycles, etc., are used to systematically gather multipleimages of the scenery surrounding the roads running throughout some partof the world, such as a given town, city, county, state or even a wholecountry. To build the virtualized view, adjacent ones of the gatheredimages are “stitched” together using known panoramic image processingtechniques. Typically this virtualized view is made available online(via the Internet).

For example, each image captured by an image-gathering car of the streetviewing service is typically linked to respective location datarecording the location at which the image was captured, e.g. in terms ofGPS coordinates, and this location data may in turn be linked to a mapof an online mapping service. This allows the virtualized view to bemade accessible through the mapping service as a front-end. The user canthen access the virtualized view by accessing the mapping servicethrough a browser or a dedicated app running on his or her smartphone,tablet, laptop or desktop computer, or the like. E.g. the user may dragand drop an icon onto a particular position on a street shown on anonline map, and in response the online service then presents the userwith a navigable virtual view of the surroundings starting from theselected position. Further, the location data allows additionalinformation to be added in the virtualized view, for example as anoverlay image. Currently this is used for showing the user thecoordinates at which the image was taken, and for showing the streetaddress or name of a building visible in the image.

In various jurisdictions, privacy and other regulations apply requiringimages of, for example, license plates and faces to be blurred. Hence tocomply with various privacy regulations in such jurisdictions, theimages are processed to automatically detect license plates and facesand blur these. Also, for state security and other reasons certainbuildings or details will not be captured or will be deleted at a laterstage based on a manual request. In addition, users can request thatelements in the virtualized view are censored or removed. An examplewould be the parents of a murdered child whose dead body was visible inthe street viewing service, or sunbathers who in their gated backyardwhere not visible from eye level but were captured by the elevatedcamera on the image-gathering car, etc.

To remove or blur objects in the virtualized view of the street viewingservice, images can be automatically analysed to detect certainpredetermined types of objects such as license plates and human faces.Also, images can be manually reviewed and censored, or images can bemarked as censored based on metadata such as the location (e.g. based oncaptured GPS coordinates), such that the service automatically obscuresor removes any images associated with the relevant metadata.

SUMMARY

However, each of these approaches has one or more disadvantages, such asmonetary cost, success rate (false positives and/or negatives), impacton the product, compliancy with local regulations, etc. For example, thecurrent automated image processing tools used to blur license plates andfaces are not perfect. Possible fines for non-compliance and the cost ofmanual processing of user requests are undesirable. Hence existingtechniques are inefficient, error prone and/or not user friendly

Nonetheless, an increased desire for privacy drives citizens to demandgreater privacy protection. At the same time the added value of streetviewing services are apparent and an increasing number of people makeuse of these services. For businesses these services offer valuablealternatives for providing information to (potential) customers.

It would therefore be desirable to provide a more advanced method forthe processing of image data captured in public spaces, or the like.

According to one aspect disclosed herein, there is provided an imageprocessing device (e.g. a server) comprising a stitching moduleconfigured to receive a first image of a first region of a physicalspace, and a second image of a second region of said physical space, andto determine a spatial relationship between the first image and thesecond image, and thereby stitch together the first and second imagesinto a composite image comprising a first area derived from the firstimage and a second area derived from the second image (N.B. the firstand second areas will usually partially overlap). The image processingdevice also comprises a signal detection module configured to detect anembedded signal having been embedded in light illuminating at least partof the first region of said physical space upon capture of the firstimage, the embedded signal conveying metadata relating to at least partof said physical space. Preferably the signal detection module does thisby detecting the embedded signal from the first image itself (i.e. thesignal is captured in the first image and extracted therefrom), thoughalternatively the signal may be detected by using a separate lightsensor operating in association with the capture of the first image.Either way, the signal detection module is configured to apply an effectto at least part of the first area of the composite image based on thedetected metadata, and also, based on the detected metadata and on thedetermined spatial relationship determined between the first and secondimages, to apply the effect to at least part of the second area of thecomposite image that extends beyond the first area (i.e. a part thatdoes not overlap with the first area).

The invention thus provides a user friendly way for providing metadatarelated to a physical space captured by an image processing device,allowing for one or more advantages such as more efficient dataprocessing, greater control for the user, and/or lower failure rate inapplying the metadata. The system uses a light communication techniquesuch as coded light to transmit metadata relating to a physical space.This can be beneficial for one or more of a number of reasons. Forexample, it can be beneficial compared to using radio frequency beacons(e.g. Bluetooth, Wi-Fi, RFID or the like) because it causes less RFpollution, and it can also be beneficial compared to using overtlyvisible marks (e.g. a QR code) because it can require lower resolutionimage capturing and/or can be less dependent on the position of thevisible mark to the image capturing device. Also, in embodiments codedlight can be captured by the same camera that is used to capture images,avoiding the need for additional sensors.

In embodiments, the effect applied based on the metadata may compriseremoving or obscuring (e.g. blurring) said parts of the first and secondareas of the composite image. Alternatively said effect may comprisespreventing the composite image being stored, transferred or copied if itcontains any of said parts of the first and second areas.

As another alternative, the effect need not be for privacy, but ratherto provide information to the viewer or otherwise enhance the compositeimage. Hence said effect may comprise overlaying a graphical elementover said parts of the first and second areas of the composite image,and/or overlaying a graphical element over the composite image inassociation with said parts of the first and second areas. E.g. theeffect could comprise a message such as a “for sale” sign or a name of abusiness overlaid over or nearby the relevant areas of the compositeimage, and/or may comprise highlighting the areas in question.

In embodiments, the signal detection module may be configured to detecta position of a light source generating said light, and based on thedetected position of the light source and on the determined spatialrelationship between the first image and the second image, to determinean extent to which the metadata applies within the composite image andthereby determine said parts of the first and second areas.

For example, the metadata may specify that the image processing moduleis to obscure or remove parts of the composite image showing anythingwithin a specified radius from the position of the light source withinsaid physical space (e.g. blur all objects and other scenery within 10 mor 20 m radius of the position of the light source). In this case, theimage processing module is configured to determine said extent in thecomposite image (i.e. the extent to which the metadata applies) as thatcorresponding to the specified radius from the position of the lightsource within said physical space.

In embodiments, the stitching module is configured to determine saidspatial relationship between the first and second images based on aspatial relationship between first and second regions of the physicalspace. For example, the stitching module may be configured to determinethe spatial relationship between the first and second regions of thephysical space based on a positions at which the first and second imageswere captured as detected by a positioning system at a time of thecapture (e.g. based on GPS coordinates captured by the camera or thecamera-carrying vehicle).

Alternatively or additionally, the stitching module may be configured todetermine said spatial relationship between the first and second imagesbased on image recognition applied to the first and second images (i.e.by matching parts of recognizable objects in the first and second imagesto one another).

In a particularly preferred application, the image processing device maybe arranged to make the composite image including said effect availableonline as part of a street viewing service.

In further embodiments, the stitching module and/or signal detectionmodule, as appropriate, may be configured to perform operations inaccordance with any of the image processing features disclosed herein.

According to another aspect disclosed herein, there is provided a systemcomprising the image processing device according to any embodimentdisclosed herein, and also a camera configured to capture the first andsecond images. In embodiments, the camera may be mounted on a vehiclesuch as a car, which travels about and systematically gathers images ofits surroundings including the first and second images, and recordsrespective locations (e.g. GPS coordinates) at which those images werecaptured as it goes. The gathered images and associated locations canthen be used by the stitching module to create the composite image.

According to another aspect disclosed herein, there is provided a methodcomprising: receiving a first image of a first region of a physicalspace, and a second image of a second region of said physical space;determining a spatial relationship between the first image and thesecond image, and thereby stitching together the first and second imagesinto a composite image comprising a first area derived from the firstimage and a second area derived from the second image; detecting anembedded signal having been embedded in light illuminating at least partof the first region of said physical space upon capture of the firstimage, the embedded signal conveying metadata relating to at least partof said physical space; and applying an effect to at least part of thefirst area of the composite image based on the detected metadata, andalso, based on the detected metadata and on the determined spatialrelationship between the first and second images, applying the effect toat least part of the second area of the composite image that extendsbeyond the first area.

According to another aspect disclosed herein, there is provided acomputer program product comprising code embodied on a computer-readablestorage medium and/or being downloadable therefrom, the code beingconfigured so as when run on a signal processing device to performoperations of: receiving a first image of a first region of a physicalspace, and a second image of a second region of said physical space;determining a spatial relationship between the first image and thesecond image, and thereby stitching together the first and second imagesinto a composite image comprising a first area derived from the firstimage and a second area derived from the second image; detecting anembedded signal having been embedded in light illuminating at least partof the first region of said physical space upon capture of the firstimage, the embedded signal conveying metadata relating to at least partof said physical space; and applying an effect to at least part of thefirst area of the composite image based on the detected metadata, andalso, based on the detected metadata and on the determined spatialrelationship between the first and second images, applying the effect toat least part of the second area of the composite image that extendsbeyond the first area.

In embodiments, the method may comprise operations or the computerprogram may be configured to perform operations in accordance with anyof the image processing features disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

To assist understanding of the present disclosure and to show howembodiments may be put into effect, reference is made by way of exampleto the accompanying drawing in which:

FIG. 1 is a schematic illustration of a system for capturing andstitching images,

FIG. 2 is a schematic illustration of two individually captured imagesof a scene,

FIG. 3 is a schematic illustration of a composite image stitchedtogether from two individually captured images,

FIG. 4 is a schematic illustration of a composite image blurred based onmetadata detected in relation to the image, and

FIG. 5 is a schematic illustration of a composite image with informationoverlaid based on metadata detected in relation to the image.

DETAILED DESCRIPTION OF EMBODIMENTS

The following describes a system and method which uses coded light,emitted towards an image capturing device, to provide the imagecapturing device with metadata (such as privacy settings). This metadatais then used in the processing of the captured images. However, a singlescene (e.g. the left side of a particular road) is not captured in asingle image. Rather, multiple smaller images are taken and these arethen ‘stitched’ to create a larger image encompassing the scene. Toaccommodate this, the method comprises: (i) capturing first image data,comprising coded light, of a first region in a physical space, (ii)capturing second image data of a second region, different from the firstregion (and which may be partially overlapping), in the physical space;(ii) determining a spatial relationship between the first and the secondimage data based on a spatial relationship between the first and thesecond region (iv) extracting, from the coded light, metadata related toat least the second region; and (v) processing at least the second imagedata based on the extracted metadata and the determined spatialrelationship between the first and the second region.

FIG. 1 illustrates a system in accordance with embodiments of thepresent disclosure. The system comprises three components to worktogether: (a) a lighting device 102 for emitting coded light; (b) animage capturing device 104 for capturing photographic images; and (c) animage processing device 106 for processing the images, and inembodiments making them available as part of a street viewing service.

The lighting device 102 is deployed as part of a scene 100 (i.e. aphysical space) comprising one or more objects 101 with which a userwishes to associate metadata. For instance, the one or more objects 101may comprise one or more buildings, and the user may be an owner of thebuildings. As will be discussed in more detail shortly, the metadata isincluded by means of a signal embedded in the light emitted by thelighting device 102. The metadata may comprise an indication that theuser wishes a certain degree of privacy to be respected by a streetviewing service, e.g. wants the building(s) 101 to be blurred whenappearing in the street viewing service. Alternatively the metadata maycomprise information that the user wishes to be shown in associationwith the building(s) or object(s) 101 when viewed through the streetviewing service, e.g. the fact that a building 101 is for sale.

The image capturing device 104 takes the form of a camera unit mountedon a vehicle 105 driven by a human or computer operator. E.g. thevehicle may take the form of an automobile such as a car, van,motorcycle; or may take the form of a manually powered vehicle such as abicycle or handcar. Alternatively the camera unit 104 may be carriedabout the person of a human operator, either riding in a vehicle 105 oron foot. The vehicle 105 and/or human operator may be part of anoperation run or commissioned by a provider of the street viewingservice. In this case the vehicle 105 or operator will typically beinstructed to systematically traverse a system of roads in theenvironment of interest (e.g. in a certain town or city). During thissystematic journey, the camera 104 is arranged to capture multipleimages (photographs) at different respective positions along each road,and to submit these images to the street viewing service as it/he/shegoes. Alternatively, the operator of the camera 104 could be a userhaving no particular association with the street viewing service, butwho voluntarily submits images on an ad hoc basis (if enough users dothis, the street viewing service can gradually build up a sufficientdatabase of images).

Either way, the images are submitted to an image processing device 106in the form of a server of the street viewing service (note that aserver as referred to herein refers to a logical or functional entitythat may comprise one or more server units at one or more geographicalsites). Further, in embodiments the camera 104 (or an accompanyingdevice working in conjunction with it) is equipped with a client of apositioning system, arranged to use the positioning system to detect thelocation at which each of the images was captured. For example, thepositioning system may be a satellite based positioning system such asGPS (Global Positioning System). The camera 104 (or accompanying device)is thereby arranged to record in association with each image arespective geographical position at which each respective image wascaptured, and to submit this to the server 106 in association with eachrespective image.

The server 106 comprises a stitching module 108 arranged to stitchtogether adjacent ones of the images into a panoramic whole, based onpanoramic image processing techniques which are in themselves known inthe art. In embodiments, the stitching module is configured to do thisbased on the geographic locations of the images as collected at the timeof capture (e.g. GPS coordinates). Alternatively or additionally, thestitching could be performed based on object recognition techniques,i.e. if the stitching module 108 recognises one side or portion of anobject and another (preferably overlapping) side or portion of the sameobject in another image, this recognition can be used to stitch togetherthe adjacent images containing those portions of the object. Thestitching module 108 may be implemented in software stored on one ormore memory devices of the server 106 (e.g. a magnetic memory devicesuch as a hard disk or an electronic storage device such as a flashmemory) and arranged to run on one or more processors of the server 106(or more generally the image processing device). Alternatively it is notexcluded that the stitching module 108 could be implemented in dedicatedhardware circuitry, or configurable or reconfigurable hardware (e.g. PGAor FPGA), or any combination of hardware and software.

The server 106 is arranged to make the resulting panorama created by thestitching available to the public in the form of a virtualized view,accessible via a wide area network such as the Internet 112. That is, auser of a user terminal 114 such as a smartphone, tablet, laptop ordesktop computer can connect to the server 106 via the Internet 112(e.g. using a web browser or dedicated app running on the user terminal114), and thereby view and preferably navigate their way through thepanoramic virtual view created as a result of the stitching performed bythe stitching module 108. For example, the provider of the streetviewing service may also be the provider of an online map service,making available a road map of a town, city, country, state, country oreven the whole world via the Internet 112. In this case, a user runs aweb browser or dedicated app on his or her user terminal 114, whichoperates the user terminal 114 to access the online mapping service fromthe server 106 via the Internet 112. The user terminal 114 thus displaysthe map (or a portion of it) to the user. The user is then able toselect a particular point on the map, e.g. a particular point along aparticular road. In response, the mapping service links to the streetviewing service with the selected location, and the street viewingservice provides the user terminal 114, via the Internet 112, with thevirtual view corresponding to that location, to be displayed via ascreen of the user terminal 114. The user may then use a user interfaceon the user terminal 114 to navigate the virtual view, such as bychanging the angle, moving to a different position, and/or zooming in orout.

Thus based on the system outlined above, the camera 104 will capturemultiple images of the physical location surrounding it (or on a singleside of the road, etc.), with each image relating to a region in aphysical space. For example an image is taken every meter that the car105 moves forward. There will most likely be overlap between the images,which is advantageous for the image processing step as it allows forbetter stitching of the individual images to create a larger imagecovering multiple regions. If no information is available on how far thecapturing device 104 has moved in between taking pictures, then spatialdata could be extracted from the images based on at least theoverlapping part of these images. Either way, the resulting virtualizedview can then be made available to users in the form of a street viewingservice or the like.

With reference to FIG. 2, consider for the sake of illustration a scene100 (i.e. a physical space) comprising one or more buildings 101,wherein the scene 100 is captured by the camera 104 over two (or more)different images 200 a, 200 b taken from different positions such thatthe images 200 a, 200 b overlap to some extent but are not completelycoincident. For example, the scene 100 may comprise a building, or acomplex of buildings 101 such as a school, owned or leased by a givenparty, or being the site of an enterprise run by a given party. However,despite the fact that the building or complex 101 may fall within theownership, responsibility or interest of a single party, it is largeenough that it appears across multiple images 200 a, 200 b captured bythe street viewing service, i.e. the building or complex 101 as a wholedoes not fall completely within any given one of the images 200 a, 200b.

Nonetheless, as shown in FIG. 3, the different adjacent images 200 a,200 b are stitched together by the stitching module 108 into a compositeimage 300, comprising a first area 300 a corresponding to the firstcaptured image 200 a and a second area 300 b corresponding to the secondcaptured image 300 b (note that the first and second areas 300 a, 300 bof the composite image preferably overlap so as to assist stitching).Therefore the whole building or complex 101 does appear over thecomposite image 300, and will be visible to the public in this formthrough the street viewing service. It may be that a party having aninterest in the building or complex 101 (e.g. the owner) wishes to applymetadata in relation to the image of the building or complex 101, andfor this to apply across all the parts of the composite image 300showing the building or complex 101 in question, not just the part in anindividual one images 200 a. For instance, the party may wish for thewhole building or complex 101 to be censored (e.g. blurred) in thestreet viewing service, or may wish to highlight the whole building orcomplex 101 as being for sale.

According to embodiments of the present disclosure, the party wishing toapply the metadata deploys a lighting device 102 in the scene 100,either attached to, inside, or in the vicinity of the building orcomplex 101 in question. For example the lighting device 102 could be anoutdoor luminaire fixed to an exterior surface of the building orcomplex 101, or a fixed or free-standing luminaire placed in the groundsof the building or complex 101, or even a luminaire located inside thebuilding or complex but visible through a window or other opening. Thelighting device 102 is configured to include an embedded signal in thevisible light it emits, the signal comprising metadata relating to thebuilding or complex 101. The lighting device 102 may be part of ageneral purpose illumination system with a primary purpose ofilluminating the building or complex 101 or the scene 100, such that theembedded signal is embedded in the general purpose illumination.Alternatively the lighting device 102 may be a dedicated device with adedicated purpose of emitting a signal by means of light.

Lighting devices for emitting embedded signals are in themselves knownthe art. One example is to use coded light, whereby the intensity of thelight emitted by the light source 102 is modulated at a frequency highenough to be invisible to the human eye (or at least tolerably so). Inembodiments, the modulation may comprise a single tone (sinusoid) or asingle oscillating waveform (e.g. rectangular wave) and the frequency ofthis tone or waveform acts as an embedded code (i.e. different lightsources each emit light with a different unique modulation frequency,unique within the system in question). This code can then be mapped to apredetermined meaning by means of a look-up performed at the receiveside. Alternatively more complex modulation schemes are possible inorder to embed more complex data. For example the modulation frequencymay be varied to represent data according to a frequency keying scheme,or the phase of the modulation frequency may be varied to represent dataaccording to a phase keying scheme, or the amplitude of the modulationmay be varied to represent data according to an amplitude keying scheme(e.g. a Manchester code). In such cases the information to be conveyedcan be embedded explicitly into the light, or again can be looked up atthe receive side based on a received code.

Other forms of embedding a signal into light may comprise using thespectral composition of the light to convey information, or embedding anunobtrusive spatial pattern into the illumination illuminating thebuilding or complex 101 or the scene 100.

By any such means, the lighting device 102 can thus be configured toemit light embedded with a signal communicating any desired metadatarelating to a physical location 100, such as a particular building orcomplex 101 at that location. Such metadata can be privacy related datasuch as what privacy setting a user has selected, e.g.: blur image datashowing people or people's faces, or do not capture any image withinthree meters of the light source, or blur 20 meters around the lightsource (=BL⁺ 20_CTR). Alternatively the metadata may comprise other datasuch as to provide information relating to the scene 100, e.g. “propertyfor sale”, “yard sale coming up on January 2nd”, or “the names of thestores located in this building are ‘Hugh's Pharmacy’, ‘Super Burgers,[etc.]”).

The image capturing device 104 may take any suitable form known in theart. It is configured to both capture images 200 a, 200 b of the scene100 as it passes, and also to capture the coded light, preferably aspart of the captured image itself. For example, the camera 104 may takethe form of a rolling-shutter camera in which the pixels of the imagesensor are grouped into a plurality of lines (e.g. horizontal rows), andthe camera 104 captures an image by exposing each of the lines in asequence, line-by-line at slightly different successive times. Thus eachline captures the light from the light source 102 at a slightlydifferent time, and hence a different phase of the modulation. If theline rate is high enough relative to the modulation frequency, thistherefore enables the modulation to be detected in the image 200 a. Ifthe code is short enough relative to the number of lines in a frame,then the code can be detected in a single frame; or otherwise the codecan be detected over multiple frames of a video image. Also, if thecamera is not a rolling shutter camera but rather a global shuttercamera which exposes the whole frame at once, then the coded light canstill be detected from a video image if the frame rate is high enoughrelative to the modulation frequency. Suitable coded light techniqueswill in themselves be familiar to a person skilled in the art.

The image processing module 106 comprises a coded light detection module110 arranged to extract the coded light signal from the images capturedand submitted by the camera 104. The coded light detection module 110may be implemented in software stored on one or more memory devices ofthe server 106 (e.g. a magnetic memory device such as a hard disk or anelectronic storage device such as a flash memory) and arranged to run onone or more processors of the server 106 (or more generally the imageprocessing device). Alternatively it is not excluded that the codedlight detection module 110 could be implemented in dedicated hardwarecircuitry, or configurable or reconfigurable hardware (e.g. PGA orFPGA), or any combination of hardware and software.

Once the coded light has been extracted, the metadata and relatedactions can be determined. In embodiments, this will comprise modifyingthe image 300 as made available through the street viewing service. Forexample, the metadata may cause the view to be blurred around a certainbuilding or complex, or a certain radius around the light source 102.

Thus the image capturing device 104 of the street-viewing service willcapture the coded light metadata appearing in the first image 200 a, andadapt the view 300 made available to the public accordingly.

In the example shown, the light source 102 appears in the first image200 a. It is known to use coded light as a means for watermarking ascene, so any given image capturing the coded light will contain thewatermark (e.g. in previous applications, so that unauthorized imagescan be forensically identified later). However, it has not previouslybeen recognized that coded light detected in a first image 200 acovering a first region of a scene 100 may in fact also be intended tocover a second region of a scene 100 appearing only in a second image200 b, even though the coded light may not necessarily quite bedetectable in the second image 200 b.

According to embodiments herein, this is addressed by using the spatialrelationship determined by the stitching module 106 in order to applythe coded light metadata detected in one image 200 a to another,adjacent image 200 b. That is, the coded light detection module 110 isconfigured to modify the composite image 300 to apply some effect 401(e.g. see FIGS. 4 and 5) based on the metadata detected from the firstcaptured image 200 a, wherein this effect is applied not just to thearea 300 a of the composite image 300 derived from the first capturedimage 200 a (or a part of it), but also the area 300 b of the compositeimage 300 derived from the second captured image 200 b (or again to apart of it).

As discussed, the image capturing device 104, such as a camera unitmounted on a car 105 of the street viewing service, will capturemultiple images of the physical location surrounding it (or on a singleside of the road, etc.). Each image relates to a region in the physicalspace, for example an image is taken every meter that the car movesforward, and there will most likely be overlap between the images, whichis advantageous to assist in the ‘stitching’ of the individual images200 a, 200 b to create a larger image 300 covering multiple regions.

As part of capturing the images, the coded light emitted by the lightingdevice 102 is also captured. In various scenarios, the lighting device102 may be captured directly in the image 200 a (i.e. the source of thecoded light can be pinpointed), or the coded light may be directlyreceived yet the lighting device 102 is not visible (e.g. the lightingdevice is located in a room and the light exits through a window towardsthe camera 104), or the coded light may be indirectly received (i.e. thelight reflects off an object towards the camera 104).

The image processing device 106 now has multiple images related tovarious regions. In the process of capturing at least one image 200 a,coded light has been captured (note that the capturing of the codedlight can be part of the same step as capturing the image 200 a or itcan be a separate step, e.g. at the same moment, just before or afterthe image 200 a is captured; and although the same sensor, i.e. thecamera 104, can be used, this is not necessarily required). The metadatais extracted from the coded light and can be used in processing multipleof the images 200 a, 200 b. The image processing device(s) 106 will usethe images 200 a, 200 b taken and the metadata extracted from the codedlight, and determine a relationship between a first and a second image200 a, 200 b. The first image 200 a relates to a first region of aphysical space wherein coded light data is captured (either as part oftaking this same image or in a separate step). The second image relatesto a second region. There is a spatial relationship between the firstand the second region, and by determining this relationship a spatialrelationship between the first and second images 200 a, 200 b is alsodetermined. The coded light data captured in the first region can thenbe applied to the second (or further) image(s) 200 b.

The metadata embedded in the coded light emitted by the light source canrefer to various types of requests or information. For example, themetadata may specify that the image (or parts of it) should be blurred,e.g. “blur this house”, or “BL_20_CTR” (=blur 20 meters around lightsource”; instructing the image processing device 106 to determinelocation of light source 102 in stitched image, determine the radiusbased on spatial information, and blur within 20 meters of the lightsource 102). Alternatively the metadata may specify that information isto be displayed in association with the image (or parts of it), e.g.“for sale”, “show business name on street viewing service”, or addressdata such as “lower floor=Main street 5 a; upper floor=Main street 5 b”.

Given the limited bandwidth of coded light applications, suchinformation does not need to be directly encoded but instead themetadata encoded can be a reference to a website or an online servicesthat links to this data. The link to the metadata may itself comprise alink to further information, for example allowing to link interiorphotos to the exterior view of a building.

In embodiments, the metadata is linked to the first image 200 a and thefurther processing takes place at a later stage, after capture, byanother device such as the server 106 which has more computing powerthan the camera 104. Alternatively, the metadata may even cause thecapturing device (camera) 104 to directly execute the request.

As an example use case, a house may be captured in multiple images 200a, 200 b that are stitched together to create a single image 300 whichthen comprises the complete home. The coded light has been captured foronly a single image 200 a, yet as the images 200 a, 200 b are spatiallyrelated (and this is the basis for the stitching) the processing isapplied across the two (or more) images 200 a, 200 b. E.g. the codedlight emitted by the lighting device 102 is extracted from the firstimage 200 a and specifies that the house should be be blurred. As partof the image processing, the house is recognized in the composite image300 and the blurring is performed for those parts of the image 300 thatcomprise the house, spanning the parts of the house shown in both theareas 300 a, 300 b derived from both the first and second capturedimages 200 a, 22 b respectively.

A more complex example is illustrated in FIG. 4. Here, the metadata doesnot specify that the house or complex 101 per se should be blurred, butrather states that a region corresponding to a specified radius of, say,10 meters around the coded light source 102 should be blurred. Nowspatial information is needed to determine which pixels relate to thiszone of 10 meters. For example, the street viewing service may capturesuch information using laser based distance measurements and tag thisinformation to the images 200 a, 200 b captured; or other options toextract such spatial information include using stereoscopic images withthe two cameras at a known distance apart and at a known angle, ordetermining dimensions of an object in the image based on known sizes ofspecific objects recognized (e.g. the width of a door), or any othersuitable image recognition techniques. Based on this information, thecoded light detection module 110 on the image processing device 106(e.g. server) applies the blurring effect 401 to the parts of thestitched image 300 that correspond to (an approximation of) a radius of10 meters from the light source 102 in the physical space 100. If thelighting device 102 is not visible in the image 200 a, 300 (e.g. it ishidden and/or only a reflection of the light is captured), additionalprocessing could be needed. For example the approximate position of thelighting device 102 can be determined based on the direction of thelight captured, shading or fading of the light being visible in theimage, etc.

Another example is illustrated in FIG. 5. Here, the metadata specifiesinformation to be displayed in relation to a certain building. In thiscase the effect 401 to be added to the composite image 300 comprisesoverlaying the information over the image. For example, the effect 401may comprise highlighting the building in question within the compositeimage 300, and/or adding a message such as “FOR SALE”. For example forthe purpose of the highlighting, the bounds of the building could bespecified by the metadata in terms of geographical coordinates withinthe physical space 100, and the coded light detection module 110 mayconvert these to coordinates in the composite image 300. Similarly, forthe purpose of the message the metadata may specify a position fordisplaying this message, e.g. relative to the physical space 100, suchas a certain set of geographical coordinates, and these may be convertedto the coordinates of the image. Alternatively the metadata may specify“apply to this building”, and the coded light detection module 110 mayadditionally apply image recognition techniques to recognise thebuilding closest to the source 102 of the coded light, and by this meansapply the highlighting and/or message at the appropriate place withinthe image 300.

It will be appreciated that the above embodiments have been describedonly by way of example.

For instance, the applicability of the present disclosure is not limitedto street viewing services. Also, the image processing device 106 is notlimited to being implemented in a server. E.g. the techniques disclosedherein could also be used to apply metadata to panoramic images createdon a personal computer unit, such as a personal camera unit or asmartphone or tablet by which the constituent captured images arecaptured, or a desktop or laptop computer to which captured images aredownloaded. In these cases, the image processing device 106, includingthe stitching module 108 and coded light detection module 110, may beimplemented on the personal computer unit, rather than a server 106.Alternatively even if the panoramic image is for personal use, theprocessing could be offloaded to an image processing device 106implemented on a server, but as a personal panoramic image processingservice for a user's own personal images rather than as part of a publicstreet viewing service.

Further, the metadata may specify other types of censorship effect, notjust blurring or other such forms of obscuration. For example, themetadata may specify that an image should not be allowed to be capturedat all if it contains any part to which the metadata privacy settingrelates, such that any image 200 a, 200 b, 300 containing any part towhich the metadata relates (e.g. any part of a certain building orcomplex 101) is not allowed to be stored in any non-volatile form uponcapture and instead should be immediately deleted. In this case theimage processing device 106 is implemented as part of the capturingcamera unit 104; and the coded light detection module 110 of the imageprocessing device 106 is arranged to automatically act on the privacysetting as specified by the metadata, by immediately deleting theoffending image from whatever temporary logic (e.g. RAM, registers orlatches) that it is temporarily held in.

As another example, the metadata may specify that any image 200 a, 200b, 300 containing any part to which the metadata relates (e.g. any partof a certain building or complex 101) is not allowed to be transferredto any other, external destination such as another, external computerunit (or is not allowed to be transferred to one or more particularspecified prohibited destinations, or is only allowed to be transferredto one or more particular specified approved destinations). In thiscase, the image processing device 106 may again be implemented as partof the capturing camera unit 104, or any other computing unit that mightpotentially receive and disseminate such an image (e.g. a separate userterminal or a server); and the coded light detection module 110 of theimage processing device 106 is arranged to automatically act on theprivacy setting as specified by the metadata, by automatically blockingthe offending image from being received and/or forwarded.

As yet another example, the metadata may specify that any image 200 a,200 b, 300 containing any part to which the metadata relates (e.g. anypart of a certain building or complex 101) is not allowed to be uploadedto any social media service or other such public service (or is notallowed to be uploaded to one or more particular specified prohibitedservices, or is only allowed to be uploaded to one or more particularspecified approved services). In this case, the image processing device106 may again be implemented as part of the capturing camera unit 104,or any other computing unit that might potentially receive anddisseminate such an image (e.g. a separate user terminal), or any serverthat hosts a public service that might potentially publish the image;and the coded light detection module 110 of the image processing device106 is arranged to automatically act on the privacy setting as specifiedby the metadata, by automatically blocking the offending image frombeing uploaded and/or accepted for publication.

Further, the scope of the present disclosure is not limited to applyingmetadata in relation to a building or complex of buildings 101. Ingeneral the metadata may relate to any scene (any physical space) whichmay comprise for example one or more particular buildings, people,plants, art installations, and/or exhibits, etc.

Further, the applicability of the teachings herein is not limited to twoadjacent images. In general, the spatial relationship between any two ormore images may be used to take the metadata from any embedded lightsignal detected in association with one image and to apply that metadatato any other one or more spatially related images or part(s) thereof.

Furthermore, the scope of the disclosure is not limited to still images,and in other embodiments analogous techniques may be used to stitchtogether adjacent video images, and apply light-based metadata detectedfrom one such image to another.

Other variations to the disclosed embodiments can be understood andeffected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. A single processor or other unit may fulfil thefunctions of several items recited in the claims. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measures cannot be used toadvantage. A computer program may be stored/distributed on a suitablemedium, such as an optical storage medium or a solid-state mediumsupplied together with or as part of other hardware, but may also bedistributed in other forms, such as via the Internet or other wired orwireless telecommunication systems. Any reference signs in the claimsshould not be construed as limiting the scope.

1. An image processing device comprising: a stitching module configuredto receive a first image of a first region of a physical space and asecond image of a second region of said physical space, and to determinea spatial relationship between the first image and the second image, andthereby stitch together the first and second images into a compositeimage comprising a first area derived from the first image and a secondarea derived from the second image; and a signal detection moduleconfigured to detect an embedded signal having been embedded in lightilluminating at least part of the first region of said physical spaceupon capture of the first image, the embedded signal conveying metadatarelating to at least part of said physical space; wherein the signaldetection module is configured to apply an effect to at least part ofthe first area of the composite image based on the detected metadata,and also, based on the detected metadata and on the determined spatialrelationship determined between the first and second images, to applythe effect to at least part of the second area of the composite imagethat extends beyond the first area.
 2. The image processing device ofclaim 1, wherein the signal detection module is configured to detectsaid embedded signal from the first image.
 3. The image processingdevice of claim 1, wherein the signal detection module is configured todetect a position of a light source generating said light, and based onthe detected position of the light source and on the determined spatialrelationship between the first image and the second image, to determinean extent to which the metadata applies within the composite image andthereby determine said parts of the first and second areas.
 4. The imageprocessing device of claim 1, wherein said effect comprises obscuring orremoving said parts of the first and second areas of the compositeimage.
 5. The image processing device of claim 4, wherein said effectcomprises obscuring said parts of the first and second areas of thecomposite image by blurring said parts.
 6. The image processing deviceof claim 3, wherein the metadata specifies that the image processingmodule is to obscure or remove parts of the composite image showinganything within a specified radius from the position of the light sourcewithin said physical space, and wherein the image processing module isconfigured to determine said extent in the composite image as thatcorresponding to the specified radius from the position of the lightsource within said physical space.
 7. The image processing module ofclaim 1, wherein said effect comprises preventing the composite imagebeing stored, transferred or copied if it contains any of said parts ofthe first and second areas.
 8. The image processing device of claim 1,wherein said effect comprises overlaying a graphical element over saidparts of the first and second areas of the composite image, and/oroverlaying a graphical element over the composite image association withsaid parts of the first and second areas.
 9. The image processing deviceof claim 1, wherein the stitching module is configured to determine saidspatial relationship between the first and second images based on aspatial relationship between first and second regions of the physicalspace.
 10. The image processing device of claim 9, wherein the stitchingmodule is configured to determine the spatial relationship between thefirst and second regions of the physical space based on a positions atwhich the first and second images were captured as detected by apositioning system at a time of the capture.
 11. The image processingdevice of claim 1, wherein the stitching module is configured todetermine said spatial relationship between the first and second imagesbased on image recognition applied to the first and second images. 12.The image processing device of claim 1, wherein the image processingdevice is arranged to make the composite image including said effectavailable online as part of a street viewing service.
 13. A systemcomprising the image processing device of claim 1, and a cameraconfigured to capture the first and second images.
 14. A methodcomprising: receiving a first image of a first region of a physicalspace, and a second image of a second region of said physical space;determining a spatial relationship between the first image and thesecond image, and thereby stitching together the first and second imagesinto a composite image comprising a first area derived from the firstimage and a second area derived from the second image; detecting anembedded signal having been embedded in light illuminating at least partof the first region of said physical space upon capture of the firstimage, the embedded signal conveying metadata relating to at least partof said physical space; and applying an effect to at least part of thefirst area of the composite image based on the detected metadata, andalso, based on the detected metadata and on the determined spatialrelationship between the first and second images, applying the effect toat least part of the second area of the composite image that extendsbeyond the first area.
 15. A computer program product comprising codeembodied on a computer-readable storage medium and/or being downloadabletherefrom, the code being configured so as when run on a signalprocessing device to perform operations of: receiving a first image of afirst region of a physical space, and a second image of a second regionof said physical space; determining a spatial relationship between thefirst image and the second image, and thereby stitching together thefirst and second images into a composite image comprising a first areaderived from the first image and a second area derived from the secondimage; detecting an embedded signal having been embedded in lightilluminating at least part of the first region of said physical spaceupon capture of the first image, the embedded signal conveying metadatarelating to at least part of said physical space; and applying an effectto at least part of the first area of the composite image based on thedetected metadata, and also, based on the detected metadata and on thedetermined spatial relationship between the first and second images,applying the effect to at least part of the second area of the compositeimage that extends beyond the first area.